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DETAILED ACTION 



Drawings 



1. The drawings are objected to as failing to comply with 37 CFR 1 .84(p)(4) because 
reference numeral 107 has been used to designate both HMD and a player. A proposed drawing 
correction or corrected drawings are required in reply to the Office action to avoid abandonment 
of the application. The objection to the drawings will not be held in abeyance. 



2. The disclosure is objected to because of the following informalities: On page 1 1, line 4, 
reference numeral 101 is mentioned, but it is not shown in Figure 1. Appropriate correction is 
required. 



3. The following is a quotation of the appropriate paragraphs of 35 U.S. C. 102 that form the 
basis for the rejections under this section made in this Office action: 



A person shall be entitled to a patent unless - 

(e) the invention was described in a patent granted on an application for patent by another filed in the United 
States before the invention thereof by the applicant for patent, or on an international application by another who 
has fulfilled the requirements of paragraphs (1), (2), and (4) of section 371(c) of this title before the invention 
thereof by the applicant for patent. 



The changes made to 35 U.S.C. 102(e) by the American Inventors Protection Act of 1999 
(AIPA) do not apply to the examination of this application as the application being examined 
was not (1) filed on or after November 29, 2000, or (2) voluntarily published under 35 U.S.C. 
122(b). Therefore, this application is examined under 35 U.S.C. 102(e) prior to the amendment 
by the AIPA (pre-AIPA 35 U.S.C. 102(e)). 



Specification 



Claim Rejections - 35 USC § 102 
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4. Claims 1-27 are rejected under 35 U.S.C. 102(e) as being anticipated by Sato U.S. Patent 
No. 6,445,815. 

5. Claim 1 : An augmented reality presentation apparatus for superimposing a virtual object 
in a real space, characterized by comprising: augmented reality presentation means for 
superimposing the virtual object viewed from a player 's viewpoint position in the real space 
viewed from said player 's viewpoint position; the first video sensing means for sensing a video of 
the real space viewed from viewpoint position which differ from said player 's viewpoint position; 
the first video generation means for generating a video of the virtual object viewed from said 
first viewpoint position; and the first video composition means for compositing an augmented 
reality video viewed from said first viewpoint position on the basis of said videos of the real 
space and the virtual object viewed from said first viewpoint position. 

Referring to claim 1, the Sato reference discloses an augmented reality presentation 
system and apparatus that generates and presents a virtual image in a real space. The Sato 
reference also discloses an augmented reality presentation means such as an optical see-through 
head mounted display displaying a merged real objects and virtual objects (column 1, lines 13- 
23). The Sato reference teaches a sensing means such as two cameras 102R and 102L along with 
a position/posture sensor 101 for sensing a scene of a real space (column 5, lines 1-53). The Sato 
reference also teaches a video generation means such as an image generation apparatus 200 to 
continuously generate a depth image on the basis of an image from one camera or depth images 
on the basis of images from two cameras (column 7, lines 10-14) and a video composition means 
such as a warping means for warping the first depth image generated by the depth image 



j 




Application/Control Number: 09/658,463 Page 4 

Art Unit: 2672 

generation means to a second depth image at the second viewpoint on the basis of the position 
and posture information at the second viewpoint (column 5, lines 1-17). 

Claim 2: The apparatus according to claim 1, characterized in that said augmented 
reality presentation means further comprises: the second video sensing means for sensing a 
video of the real space viewed from said player 's viewpoint position; the second video 
generation means for generating a video of the virtual object viewed from said player 's 
viewpoint position; the second video composition means for compositing an augmented reality 
video viewed from said player 's viewpoint position on the basis of said videos of the real space 
and the virtual object viewed from said player 's viewpoint position; and the display means for 
displaying to the player the augmented reality video viewed from said player 's viewpoint 
position. 

Claim 2 recites all the limitations of claim 1 and adds the limitation of "the second video 
sensing means," "the second video generation means," "the second video composition means," 
and "display means." The Sato reference teaches an augmented reality presentation means 
including a video sensing means such as HMD 100 with a three-dimensional position/posture 
sensor 101 for sensing a video of the virtual object (column 12, lines 12-27). The Sato reference 
also teaches a video generation means such as a depth image generation apparatus 200 for 
generating depth images in correspondence with the right and left viewpoints of the player 
(column 12-50). Sato further teaches a video composition means such as depth warping module 
203 and a CG renderer 302 for compositing an augmented reality video viewed from a player's 
viewpoint position (column 13, lines 32-60 and column 14, lines 1-29). Finally, Sato teaches 
display means such as LCDs 103 for displaying a rendered image (column 14, lines 1-29). 
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Claim 3 : The apparatus according to claim I, characterized in that said augmented 
reality presentation means further comprises: the second video generation means for generating 
a video of the virtual object viewed from said player 's viewpoint position; and the display means 
for displaying to the player the video of the virtual object viewed from said player 's viewpoint 
position on a display surface through which the player can visually see the real space. 

Claim 3 recites all the limitations of claim 1 and adds the limitation of "the second video 
generation means" and "the display means." The Sato reference teaches a video generation 
means such as a depth image generation apparatus 200 for generating depth images in 
correspondence with the right and left viewpoints of the player (column 12-50). The Sato 
reference further teaches display means such as LCDs 103 for displaying a rendered image 
(column 14, lines 1-29). 

Claim 4: The apparatus according to claim 7, characterized by further comprising 
information generation means for generating information that pertains to rendering of the 
virtual object, and in that said first video generation means and said second video generation 
means generate videos of the virtual object using the information that pertains to rendering of 
the virtual object 

Claim 4 recites all the limitations of claim 1 and adds the limitation of "information 
generation means" and "generation means generate videos." The Sato reference teaches an 
information generation means such as the CG renderer 302 that renders an image and depth 
image of CG data received from the three-dimensional database 301 on the basis of the 
viewpoint position/posture information (column 14, lines 1-29) and the image generation module 
300 that generates an augmented reality image using the three-dimensional CG database in 
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accordance with the distance to an object in the real world expressed by the depth image (column 
12, lines 44-50). 

Claim 5: The apparatus according to claim 4 t characterized in that said information 
generation means generates information of an outer appearance of the virtual object and 
information of a position/posture of the virtual object as the information that pertains to 
rendering of the virtual object 

Claim 5 recites all the limitations of claim 4 and adds the limitation of "generating 
information of an outer appearance of the virtual object and information of a position/posture of 
the virtual object." The Sato reference teaches that the CG renderer 302 renders an image and 
depth image of CG data received from the three-dimensional database 301 on the basis of the 
viewpoint position/posture information (column 14, lines 1-29) and that the image generation 
module 300 generates an augmented reality image using the three-dimensional CG database in 
accordance with the distance to an object in the real world expressed by the depth image (column 
12, lines 44-50). 

Claim 6: The apparatus according to claim 1, characterized in that parameters of said 
first video sensing means are known, and said first video generation means generates the video 
of the virtual object viewed from said first viewpoint position in accordance with the known 
parameters. 

Claim 6 recites all the limitations of claim 1 and adds the limitation of "the known 
parameters." The Sato reference teaches some known parameters of video sensing means such as 
the focal length of camera 102 (column 10, lines 62-67). 
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Claim 7: The apparatus according to claim 7, characterized in that some of parameters 
of said first video sensing means are variable, said apparatus further comprises measurement 
means for measuring changes of the parameters, and said first video generation means generates 
the video of the virtual object viewed from said first viewpoint position in accordance with the 
parameters measured by said measurement means. 

Claim 7 recites all the limitations of claim 1 and adds the limitation of "variable 
parameters" and "measurement means." The Sato reference teaches the relative position/posture 
of the viewpoint of the camera 102 and the estimation module 201 to estimate a change in 
relative viewpoint position of the camera 102 viewed at different times on the basis of the time- 
sequentially extracted position/posture information of the camera 102 (column 5, lines 48-54). 
The Sato reference teaches measurement means in which the position/posture sensor 101 
continuously outputs viewpoint position/posture information of the camera 102 along a time axis 
of the sensor 101 (column 8, lines 42-47). 

Claim 8: The apparatus according to claim 7, characterized in that the parameters of 
said first video sensing means measured by said measurement means include at least one of a 
viewpoint position/posture, and zoom ratio. 

Claim 8 recites all the limitations of claim 7 and adds the limitation of "position/posture 
information." The Sato reference teaches the relative position/posture of the viewpoint of the 
camera 102 and the estimation module 201 to estimate a change in relative viewpoint position of 
the camera 102 viewed at different times on the basis of the time-sequentially extracted 
position/posture information of the camera 102 (column 5, lines 48-54). 
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Claim 9: The apparatus according to claim 1, characterized in that when a plurality of 
first video sensing means equivalent to said first video sensing means are present, said 
apparatus further comprises selection means for receiving a plurality of videos of the real space 
from said first viewpoint position from the plurality of first video sensing means, and outputting 
a video of the real space viewed from said first viewpoint position from one selected first video 
sensing means to said first video composition means, and said first video composition means 
generates a video of the virtual object viewed from said first viewpoint position using parameters 
of the first video sensing means selected by said selection means. 

Claim 9 recites all the limitations of claim 1 and adds the limitation of "selection means 
for receiving a plurality of first video sensing means." The Sato reference teaches two cameras 
102R and 102L for sensing a scene in front of them where image signals that represent an 
environment scene of a real space sensed by the respective cameras (column 5, lines 1-17) and 
depth warping module 203 generate a depth image on the basis of an image from one camera 
depending on the purposes (column 7, lines 1-14). 

6. Claim 10 : An augmented reality presentation method for superimposing a virtual object 
in a real space, characterized by comprising: augmented reality presentation step of 
superimposing the virtual object viewed from a player 's viewpoint position in the real space 
viewed from said player 's viewpoint position; the first video sensing step of sensing a video of 
the real space viewed from viewpoint position which differ from said player 's viewpoint position; 
the first video generation step of generating a video of the virtual object viewed from said first 
viewpoint position; and the first video composition step of compositing an augmented reality 
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video viewed from said first viewpoint position on the basis of said videos of the real space and 
the virtual object viewed from said first viewpoint position. 

Referring to claim 10, the Sato reference discloses an augmented reality presentation 
method that generates and presents a virtual image in a real space. The Sato reference further 
discloses an augmented reality presentation step in which an image of merged real objects and 
virtual objects are displayed in LCDs 103 in an optical see-through head mounted device 
(column 1, lines 13-23). The Sato reference teaches a sensing step of sensing a scene of a real 
space by two cameras 102R and 102L along with a position/posture sensor 101 (column 5, lines 
1-53). The Sato reference also teaches a video generation step to continuously generate a depth 
image on the basis of an image from one camera or depth images on the basis of images from 
two cameras by an image generation apparatus 200 (column 7, lines 10-14) and a video 
composition step to inversely project a depth image ID acquired at a viewpoint having 
position/posture information into a space, and to re-project it onto the imaging plane of the 
virtual camera with the focal length assumed at the viewpoint having another estimated 
position/posture value (column 13, lines 32-48). 

The Sato reference teaches in an alternative embodiment in figure 18 a program code of 
an augmented reality presentation step including a position/posture estimation module 201, an 
image generation module 300, a depth-warping module 203 and a CG renderer 302 of the third 
embodiment. The Sato reference teaches the position/posture estimation module 201 that 
estimates movement of the viewpoint on the basis of images input from the cameras 102R and 
102L (column 15, lines 38-44). Sato also teaches a depth warping module 203 that receives the 
transformation matrices representing three-dimensional motions and outputs depth images 
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corresponding to the positions/postures of right and left viewpoints (column 16, lines 21-33). 
The Sato reference also teaches an alternative image generation module 300 that merges and 
outputs real images and images from the CG Tenderer 302 (column 16, lines 35-43). The Sato 
reference further teaches that the CG renderer 302 renders a real image and depth image of CG 
data received from a three-dimensional database 301 on the basis of the viewpoint 
position/posture information of the camera input from the position/posture estimation module 
(column 16, lines 34-67) and generates the real image to be displayed and that alternatively the 
CG renderer 302 receives the warped depth image from the depth warping module 203 and real 
image from the camera and renders a real image that has the depth of the depth image and 
renders and overlays a virtual image by using ordinal depth-keying technique (column 16, lines 
52-61). 

Claim 1 1 : The method according to claim 10, characterized in that said augmented 
reality presentation step further comprises: the second video sensing step of sensing a video of 
the real space viewed from said player 's viewpoint position; the second video generation step of 
generating a video of the virtual object viewed from said player 's viewpoint position; the second 
video composition step of compositing an augmented reality video viewed from said player 's 
viewpoint position on the basis of said videos of the real space and the virtual object viewed 
from said player 's viewpoint position; and the display step of displaying to the player the 
augmented reality video viewed from said player 's viewpoint position. 

Claim 1 1 recites all the limitations of claim 10 and adds the limitation of "the second 
video sensing step", "the second video generation step", "the second video composition step" 
and "display step." The Sato reference teaches an augmented reality presentation step including a 
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video sensing step for sensing a video of the virtual object in HMD 100 along with a three- 
dimensional position/posture sensor 101 (column 12, lines 12-27). The Sato reference also 
teaches a video generation step of generating depth images in correspondence with the right and 
left viewpoints of the player in a depth image generation apparatus 200 (column 12-50). Sato 
further teaches a video composition step of compositing an augmented reality video viewed from 
a player's viewpoint position in depth warping module 203 and a CG renderer 302 (column 13, 
lines 32-60 and column 14, lines 1-29). Finally, Sato teaches display step of displaying a 
rendered image in LCDs 103 (column 14, lines 1-29). 

The Sato reference teaches in an alternative embodiment in figure 18 a program code of 
an augmented reality presentation step including a position/posture estimation module 201, an 
image generation module 300, a depth-warping module 203 and a CG renderer 302. The Sato 
reference teaches the position/posture estimation module 201 that estimates movement of the 
viewpoint on the basis of images input from the cameras 102R and 102L (column 15, lines 38- 
44). Sato also teaches a depth warping module 203 that receives the transformation matrices 
representing three-dimensional motions and outputs depth images corresponding to the 
positions/postures of right and left viewpoints (column 16, lines 21-33). The Sato reference also 
teaches an alternative image generation module 300 that merges and outputs real images and 
images from the CG renderer 302 (column 16, lines 35-43). The Sato reference further teaches 
that the CG renderer 302 renders a real image and depth image of CG data received from a three- 
dimensional database 301 on the basis of the viewpoint position/posture information of the 
camera input from the position/posture estimation module (column 16, lines 34-67) and 
generates the real image to be displayed and that alternatively the CG renderer 302 receives the 
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warped depth image from the depth warping module 203 and real image from the camera and 
renders a real image that has the depth of the depth image and renders and overlays a virtual 
image by using ordinal depth-keying technique (column 16, lines 52-61). 

Claim 12: The apparatus according to claim 10, characterized in that said augmented 
reality presentation step further comprises: the second video generation step of generating a 
video of the virtual object viewed from said player 's viewpoint position; and the display step of 
displaying to the player the video of the virtual object viewed from said player } s viewpoint 
position on a display surface through which the player can visually see the real space. 

Claim 12 recites all the limitations of claim 10 and adds the limitation of "the second 
video generation step" and "the display step." The Sato reference teaches a video generation step 
of generating depth images in correspondence with the right and left viewpoints of the player in a 
depth image generation apparatus 200 (column 12-50). The Sato reference further teaches 
display step of displaying a rendered image in LCDs 103 (column 14, lines 1-29). 

Claim 13: The method according to claim 10, characterized by further comprising 
information generation step of generating information that pertains to rendering of the virtual 
object, and in that said first video generation step and said second video generation step 
generate videos of the virtual object using the information that pertains to rendering of the 
virtual object. 

Claim 13 recites all the limitations of claim 10 and adds the limitation of "information 
generation step" and "first and second generation steps generate videos." The Sato reference 
teaches an information generation step in which the CG Tenderer 302 renders an image and depth 
image of CG data received from the three-dimensional database 301 on the basis of the 
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viewpoint position/posture information (column 14, lines 1-29) and video generation steps in 
which the image generation module 300 generates an augmented reality image using the three- 
dimensional CG database in accordance with the distance to an object in the real world expressed 
by the depth image (column 12, lines 44-50). 

Claim 14: The apparatus according to claim 13, characterized in that said information 
generation step includes the step of generating information of an outer appearance of the virtual 
object and information of a position/posture of the virtual object as the information that pertains 
to rendering of the virtual object 

Claim 14 recites all the limitations of claim 13 and adds the limitation of "generating 
information of an outer appearance of the virtual object and information of a position/posture of 
the virtual object " The Sato reference teaches an information generation step that the CG 
renderer 302 renders an image and depth image of CG data received from the three-dimensional 
database 301 on the basis of the viewpoint position/posture information (column 14, lines 1-29) 
and the image generation module 300 generates an augmented reality image using the three- 
dimensional CG database in accordance with the distance to an object in the real world expressed 
by the depth image (column 12, lines 44-50). 

Claim 15: The method according to claim 10, characterized in that parameters of said 
first video sensing means are known, and said first video generation step includes the step of 
generating the video of the virtual object viewed from said first viewpoint position in accordance 
with the known parameters. 
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Claim 15 recites all the limitations of claim 10 and adds the limitation of "the known 
parameters." The Sato reference teaches the known parameters such as the focal length of 
camera 102 (column 10, lines 62-67). 

Claim 16: The method according to claim 10, characterized in that some of parameters of 
means for sensing a video viewed from said first viewpoint position are variable, said method 
further comprises the measurement step of measuring changes of the parameters, and said first 
video generation step includes the step of generating the video of the virtual object viewedfrom 
said first viewpoint position in accordance with the parameters measured in the measurement 
step. 

Claim 16 recites all the limitations of claim 10 and adds the limitation of "variable 
parameters" and "measurement step " The Sato reference teaches the relative position/posture of 
the viewpoint of the camera 102 and the estimation module 201 to estimate a change in relative 
viewpoint position of the camera 102 viewed at different times on the basis of the time- 
sequentially extracted position/posture information of the camera 102 (column 5, lines 48-54). 
The Sato reference teaches measurement step in which the position/posture sensor 101 
continuously outputs viewpoint position/posture information of the camera 102 along a time axis 
of the sensor 101 (column 8, lines 42-47). 

Claim 17: The method according to claim 16, characterized in that the parameters of the 
means for sensing a video viewed from said first viewpoint position measured in the 
measurement step include at least one of a viewpoint position/posture, and zoon ration. 

Claim 17 recites all the limitations of claim 16 and adds the limitation of 
"position/posture information." The Sato reference teaches the relative position/posture of the 
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viewpoint of the camera 102 and the estimation module 201 to estimate a change in relative 
viewpoint position of the camera 102 viewed at different times on the basis of the time- 
sequentially extracted position/posture information of the camera 102 (column 5, lines 48-54). 

Claim 18: The method according to claim 10, characterized in that when a plurality of 
means for sensing a video viewed from said first viewpoint position are present, said method 
further comprises the selection step of receiving a plurality of videos of the real space viewed 
from a first viewpoint position from the plurality of means for sensing a video viewed from said 
first viewpoint position, and outputting the video of the real space viewed from a first viewpoint 
position input from one selected means for sensing a video of said first viewpoint position to 
means for compsiting a first viewpoint video, and said first video composition step includes the 
step of generating a video of the virtual object viewed from said first viewpoint position using 
parameters of the means for sensing a video viewed from a first viewpoint position selected in 
the selection step. 

Claim 18 recites all the limitations of claim 10 and adds the limitation of "selection step 
for receiving a plurality of videos." The Sato reference teaches a selection step in which a depth 
image is generated on the basis of an image from one camera (column 7, lines 1-14). 
7. Claim 1 9: A storage medium storing a program code for superimposing a virtual object 
in a real space when said program code is loaded by computer, characterized by comprising: a 
program code of augmented reality presentation step of superimposing the virtual object viewed 
from a player 's viewpoint position in the real space viewed from said player 's viewpoint 
position; a program code of the first video sensing step of sensing a video of the real space 
viewed from viewpoint position which differ from said player 's viewpoint position; a program 
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code of the first video generation step of generating a video of the virtual object viewed from 
said first viewpoint position; and a program code of the first video composition step of 
compositing an augmented reality video viewed from said first viewpoint position on the basis of 
said videos of the real space and the virtual object viewed from said first viewpoint position. 

Referring to claim 19, the Sato reference discloses a storage medium that stores an image 
processing program, which is implemented on a computer and continuously presents three- 
dimensional images to an observer/player, storing a program code of an augmented reality, a 
depth estimation program code, a depth image generation program code, a position/posture 
information estimation program code, a warping program code and a program code of presenting 
to the observer three-dimensional images. The Sato reference implicitly teaches a program code 
including the augmented reality presentation step of generating and presenting a virtual image in 
a real space in which an image of merged real objects and virtual objects are displayed in LCDs 
103 in an optical see-through head mounted device (column 1, lines 13-23). The Sato reference 
also implicitly teaches a position/posture estimation module 201 that outputs three-dimensional 
motions from a viewpoint position of the camera to right and left viewpoint positions of the 
player (column 12, lines 54-65). The Sato reference further teaches a depth image generation 
module 300 that uses position/posture information input from the position/posture estimation 
module 201 as that for CG rendering and which generates an augmented reality image using the 
three-dimensional CG database in accordance with the distance to an object in the real world 
expressed by the depth image and presents it on the LCDs 103 (column 12, lines 44-50) and a 
depth warping module 203 to inversely project a depth image ID acquired at a viewpoint having 
position/posture information into a space, and to re-project it onto the imaging plane of the 



Application/Control Number: 09/658,463 Page 17 

Art Unit: 2672 

virtual camera with the focal length of the virtual camera assumed at the viewpoint having an 
estimated position/posture value by the viewpoint position/posture module 201 (column 13, lines 
32-48). 

Claim 20: The medium according to claim 19, characterized in that the program code of 
the augmented reality presentation step further comprises: a program code of the second video 
sensing step of sensing a video of the real space viewed from said player 's viewpoint position; a 
program code of the second video generation step of generating a video of the virtual object 
viewed from said player 's viewpoint position; a program code of the second video composition 
step of compositing an augmented reality video viewed from said player 's viewpoint position on 
the basis of said videos of the real space and the virtual object viewed from said player 's 
viewpoint position; and a program code of the display step of displaying to the player the 
augmented reality video viewed from said player 's viewpoint position. 

Claim 20 recites all the limitations of claim 19 and adds the limitation of "a program code 
of the second video sensing step", "a program code of the second video generation step", "a 
program code of the second video composition step" and "a program code of display step." The 
Sato reference teaches a program code of an augmented reality presentation step including an 
alternative position/posture estimation module 201, an alternative image generation module 300, 
an alternative depth-warping module 203 and an alternative CG renderer 302 of the third 
embodiment. The Sato reference teaches the position/posture estimation module 201 that 
estimates movement of the viewpoint on the basis of images input from the cameras 102R and 
102L (column 15, lines 38-44). Sato also teaches a depth warping module 203 that receives the 
transformation matrices representing three-dimensional motions and outputs depth images 
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corresponding to the positions/postures of right and left viewpoints (column 16, lines 21-33). 
The Sato reference also teaches an alternative image generation module 300 that merges and 
outputs real images and images from the CG Tenderer 302 (column 16, lines 35-43). The Sato 
reference further teaches that the CG renderer 302 renders a real image and depth image of CG 
data received from a three-dimensional database 301 on the basis of the viewpoint 
position/posture information of the camera input from the position/posture estimation module 
(column 16, lines 34-67) and generates the real image to be displayed and that alternatively the 
CG renderer 302 receives the warped depth image from the depth warping module 203 and real 
image from the camera and renders a real image that has the depth of the depth image and 
renders and overlays a virtual image by using ordinal depth-keying technique (column 16, lines 
52-61). 

Claim 21 : The medium according to claim 19, characterized in that the program code of 
the augmented reality presentation step further comprises: a program code of the second video 
generation step of generating a video of the virtual object viewed from said player 's viewpoint 
position; and a program code of the display step of displaying to the player the video of the 
virtual object viewed from said player 's viewpoint position on a display surface through which 
the player can visually see the real space. 

Claim 21 recites all the limitations of claim 19 and adds the limitation of "a program code 
of the second video generation step" and "a program of the display step." The Sato reference also 
teaches an alternative image generation module 300 that merges and outputs real images and 
images from the CG renderer 302 (column 16, lines 35-43). The Sato reference further teaches 
that the CG renderer 302 renders a real image and depth image of CG data received from a three- 
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dimensional database 301 on the basis of the viewpoint position/posture information of the 
camera input from the position/posture estimation module (column 16, lines 34-67) and 
generates the real image to be displayed and that alternatively the CG renderer 302 receives the 
warped depth image from the depth warping module 203 and real image from the camera and 
renders a real image that has the depth of the depth image and renders and overlays a virtual 
image by using ordinal depth-keying technique (column 16, lines 52-61). 

Claim 22: The medium according to claim 19, characterized by further comprising a 
program code of the information generation step of generating information that pertains to 
rendering of the virtual object, and in that in the program code of said first video generation step 
and said second video generation step, videos of the virtual object are generated using the 
information that pertains to rendering of the virtual object 

Claim 22 recites all the limitations of claim 19 and adds the limitation of "a program code 
of the information generation step" and "videos of virtual objects are generated." The Sato 
reference teaches in figures 18 and 20 that the CG renderer 302 renders an image and depth 
image of CG data received from the three-dimensional database 301 on the basis of the 
viewpoint position/posture information and the image generation module 300 generates an 
augmented reality image using the three-dimensional CG database in accordance with the 
distance to an object in the real world expressed by the depth image. 

Claim 23 : The medium according to claim 22, characterized in that the program code of 
said information generation step includes the step of generating information of an outer 
appearance of the virtual object and information of a position/posture of the virtual object as the 
information that pertains to rendering of the virtual object 
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Claim 23 recites all the limitations of claim 22 and adds the limitation of "generating 
information of an outer appearance of the virtual object and information of a position/posture of 
the virtual object." The Sato reference teaches in figure 18 and 20 an information generation step 
that the CG renderer 302 renders an image and depth image of CG data received from the three- 
dimensional database 301 on the basis of the viewpoint position/posture information and the 
image generation module 300 generates an augmented reality image using the three-dimensional 
CG database in accordance with the distance to an object in the real world expressed by the 
depth image. 

Claim 24: The medium according to claim 19, characterized in that parameters of means 
for sensing said first viewpoint video are known, and the program code of said first video 
generation step includes the step of generating the video of the virtual object viewed from said 
first viewpoint position in accordance with the known parameters. 

Claim 24 recites all the limitations of claim 19 and adds the limitation of "the known 
parameters." The Sato reference teaches in particular the known parameters such as the focal 
length of each camera 102 (column 16, lines 21-33). 

Claim 25: The medium according to claim 19, characterized in that some of parameters 
of means for sensing a video viewed from said first viewpoint position are variable, the program 
code of said medium further comprises the measurement step of measuring changes of the 
parameters, and the program code of said first video generation step includes the step of 
generating the video of the virtual object viewed from said first viewpoint position in accordance 
with the parameters measured in the measurement step. 



Application/Control Number: 09/658,463 Page 2 1 

Art Unit: 2672 

Claim 25 recites all the limitations of claim 19 and adds the limitation of "variable 
parameters" and "measurement step." The Sato reference teaches measurement step in which the 
position/posture estimation module 201 estimates movement of the viewpoint on the basis of 
images input from the cameras 102R and 102L by tracking changes in coordinate value of 
feature points and the position on the real space of which is known (column 15, lines 38-67). 

Claim 26: The medium according to claim 25, characterized in that the parameters of the 
means of sensing a video viewed from said first viewpoint position measured in the measurement 
step include at least one of a viewpoint position/posture, and zoon ration 

Claim 26 recites ail the limitations of claim 25 and adds the limitation of 
"position/posture information." The Sato reference teaches in figure 18 a head mounted 
position/posture sensor 101 and camera 102. The Sato reference further teaches various schemes 
for estimating viewpoint position/posture on the basis of image input from the cameras 102R and 
102L (column 16, lines 38-67). 

Claim 27: The medium according to claim 19, characterized in that when a plurality of 
means for sensing a video viewed from said first viewpoint position are present, said medium 
further comprises a program code of the selection step of receiving a plurality of videos of the 
real space viewed from a first viewpoint position from the plurality of means for sensing a video 
viewed from said first viewpoint position, and outputting the video of the real space viewed from 
a first viewpoint position input from one selected means for sensing a video of said first 
viewpoint position to means for compsiting a first viewpoint video, and the program code of said 
first video composition step includes the step of generating a video of the virtual object viewed 
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from said first viewpoint position using parameters of the means for sensing a video viewed from 
a first viewpoint position selected in the selection step. 

Claim 27 recites all the limitations of claim 19 and adds the limitation of "selection step 
for receiving a plurality of videos " The Sato reference teaches in figure 18 a depth estimation 
module 202, a viewpoint position/posture estimation module, a depth warping module and an 
image generation module. The Sato reference further teaches a selection step in which a depth 
image is generated on the basis of an image from one camera (see also column 7, lines 1-14). 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

Claim 28-30 are rejected under 35 U.S.C. 103(a) as being unpatentable over Sato U.S. 
Patent No. 6,445,815. 

Referring to claims 28-30, the Sato reference discloses an augmented reality presentation 
system that generates and presents a virtual image in a real space (see figures 18-20 of Sato). The 
Sato reference teaches in figure 20 a video composition means that outputs display image for 
displaying on 103R and 103L. However, the reference is silent on a printing means in connection 
to the augmented reality presentation apparatus. It is common that a computer system has a 
printing means attached to them. Therefore, it would have been obvious to one having ordinary 
skill in the art to have incorporated a printing means in the augmented reality presentation 
apparatus of Sato because such construction is rather conventional. A person of ordinary skill in 
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the art would be motivated to have incorporated a printing means to the Sato's augmented reality 
presentation apparatus to further provide a paper copy of still images of the real images that have 
been displayed on the display devices of Sato. 

Conclusion 

8. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

a. Jaszlics U.S. Patent No. 6, 166,744 discloses a system for combining virtual 
images with real-world scenes. 

b. Abraham U.S. Patent No. 5,684,943 discloses a method and apparatus for creating 
virtual worlds. 

9. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (703) 605-1213. 
The examiner can normally be reached on 8:00 AM - 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mike Razavi can be reached on (703) 305-4713. The fax phone numbers for the 
organization where this application or proceeding is assigned are (703) 308-6606 for regular 
communications and (703) 308-6606 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is (703) 395-3900. 
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